{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "`Learn the Basics <intro.html>`_ ||\n",
    "`Quickstart <quickstart_tutorial.html>`_ || \n",
    "`Tensors <tensorqs_tutorial.html>`_ ||\n",
    "`Datasets & DataLoaders <data_tutorial.html>`_ ||\n",
    "`Transforms <transforms_tutorial.html>`_ ||\n",
    "**Build Model** ||\n",
    "`Autograd <autogradqs_tutorial.html>`_ ||\n",
    "`Optimization <optimization_tutorial.html>`_ ||\n",
    "`Save & Load Model <saveloadrun_tutorial.html>`_\n",
    "\n",
    "Build the Neural Network这一章超级重要\n",
    "================================\n",
    "\n",
    "Neural networks comprise of layers/modules that perform operations on data. \n",
    "The `torch.nn <https://pytorch.org/docs/stable/nn.html>`_ namespace provides all the building blocks you need to \n",
    "build your own neural network. Every module in PyTorch subclasses the `nn.Module <https://pytorch.org/docs/stable/generated/torch.nn.Module.html>`_. \n",
    "A neural network is a module itself that consists of other modules (layers). This nested structure allows for\n",
    "building and managing complex architectures easily.\n",
    "\n",
    "In the following sections, we'll build a neural network to classify images in the FashionMNIST dataset.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "这一章超级重要\n",
    "构建神经网络\n",
    "神经网络由对数据执行操作的层/模块组成。该torch.nn命名空间提供了所有你需要建立自己的神经网络的基石。\n",
    "在PyTorch每个模块的子类nn.Module。神经网络是一个模块本身，由其他模块（层）组成。这种嵌套结构允许轻松构建和管理复杂的架构。\n",
    "在以下部分中，我们将构建一个神经网络来对 FashionMNIST 数据集中的图像进行分类。\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import datasets, transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get Device for Training\n",
    "-----------------------\n",
    "We want to be able to train our model on a hardware accelerator like the GPU, \n",
    "if it is available. Let's check to see if \n",
    "`torch.cuda <https://pytorch.org/docs/stable/notes/cuda.html>`_ is available, else we \n",
    "continue to use the CPU. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "获取训练设备\n",
    "我们希望能够在 GPU 等硬件加速器上训练我们的模型（如果可用）。让我们检查一下 torch.cuda是否可用，否则我们继续使用CPU。\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cuda device\n"
     ]
    }
   ],
   "source": [
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "print('Using {} device'.format(device))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the Class\n",
    "-------------------------\n",
    "We define our neural network by subclassing ``nn.Module``, and \n",
    "initialize the neural network layers in ``__init__``. Every ``nn.Module`` subclass implements\n",
    "the operations on input data in the ``forward`` method. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "定义类\n",
    "我们通过子类化定义我们的神经网络nn.Module，并在__init__中初始化神经网络层。每个nn.Module子类都在forward方法中实现对输入数据的操作。\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NeuralNetwork(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(NeuralNetwork, self).__init__()\n",
    "        self.flatten = nn.Flatten()\n",
    "        self.linear_relu_stack = nn.Sequential(\n",
    "            nn.Linear(28*28, 512),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(512, 512),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(512, 10),\n",
    "            nn.ReLU()\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.flatten(x)\n",
    "        logits = self.linear_relu_stack(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create an instance of ``NeuralNetwork``, and move it to the ``device``, and print \n",
    "its structure.\n",
    "我们创建 的实例NeuralNetwork，并将其移动到device，并打印其结构。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NeuralNetwork(\n",
      "  (flatten): Flatten()\n",
      "  (linear_relu_stack): Sequential(\n",
      "    (0): Linear(in_features=784, out_features=512, bias=True)\n",
      "    (1): ReLU()\n",
      "    (2): Linear(in_features=512, out_features=512, bias=True)\n",
      "    (3): ReLU()\n",
      "    (4): Linear(in_features=512, out_features=10, bias=True)\n",
      "    (5): ReLU()\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "model = NeuralNetwork().to(device)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the model, we pass it the input data. This executes the model's ``forward``,\n",
    "along with some `background operations <https://github.com/pytorch/pytorch/blob/270111b7b611d174967ed204776985cefca9c144/torch/nn/modules/module.py#L866>`_. \n",
    "Do not call ``model.forward()`` directly!\n",
    "\n",
    "Calling the model on the input returns a 10-dimensional tensor with raw predicted values for each class.\n",
    "We get the prediction probabilities by passing it through an instance of the ``nn.Softmax`` module.\n",
    "为了使用模型，我们将输入数据传递给它。这将执行模型的forward以及一些后台操作。不要model.forward()直接打电话！\n",
    "\n",
    "在输入上调用模型会返回一个 10 维张量，其中包含每个类的原始预测值。我们通过将其传递给nn.Softmax模块的实例来获得预测概率。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted class: tensor([1], device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "X = torch.rand(1, 28, 28, device=device)\n",
    "logits = model(X) \n",
    "pred_probab = nn.Softmax(dim=1)(logits)\n",
    "y_pred = pred_probab.argmax(1)\n",
    "print(f\"Predicted class: {y_pred}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model Layers\n",
    "-------------------------\n",
    "\n",
    "Let's break down the layers in the FashionMNIST model. To illustrate it, we \n",
    "will take a sample minibatch of 3 images of size 28x28 and see what happens to it as \n",
    "we pass it through the network. \n",
    "-------------------------\n",
    "模型层\n",
    "让我们分解 FashionMNIST 模型中的层。为了说明这一点，我们将取一个由 3 张大小为 28x28 的图像组成的小批量样本，看看当我们通过网络传递它时会发生什么。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 28, 28])\n",
      "tensor([[[0.9410, 0.9583, 0.9018,  ..., 0.4571, 0.6299, 0.1244],\n",
      "         [0.3515, 0.5542, 0.2924,  ..., 0.7796, 0.0445, 0.3113],\n",
      "         [0.8830, 0.3692, 0.3068,  ..., 0.8959, 0.5154, 0.7514],\n",
      "         ...,\n",
      "         [0.8148, 0.7228, 0.6157,  ..., 0.3165, 0.1393, 0.0944],\n",
      "         [0.9841, 0.2654, 0.8086,  ..., 0.6165, 0.6718, 0.4353],\n",
      "         [0.2773, 0.9290, 0.4049,  ..., 0.5853, 0.6689, 0.2975]],\n",
      "\n",
      "        [[0.2232, 0.8117, 0.0302,  ..., 0.3758, 0.9702, 0.8867],\n",
      "         [0.2712, 0.3397, 0.9421,  ..., 0.4972, 0.1217, 0.5005],\n",
      "         [0.2075, 0.8968, 0.5353,  ..., 0.8885, 0.3522, 0.9779],\n",
      "         ...,\n",
      "         [0.6245, 0.0265, 0.1850,  ..., 0.1418, 0.1451, 0.8593],\n",
      "         [0.6215, 0.8064, 0.3226,  ..., 0.4022, 0.3799, 0.4896],\n",
      "         [0.3982, 0.0920, 0.9429,  ..., 0.7309, 0.8895, 0.7256]],\n",
      "\n",
      "        [[0.2465, 0.8670, 0.9637,  ..., 0.1974, 0.3719, 0.7336],\n",
      "         [0.7855, 0.9609, 0.6726,  ..., 0.7886, 0.1821, 0.8542],\n",
      "         [0.4239, 0.1069, 0.6252,  ..., 0.8060, 0.3373, 0.4316],\n",
      "         ...,\n",
      "         [0.1089, 0.3892, 0.6819,  ..., 0.4696, 0.1080, 0.3171],\n",
      "         [0.5526, 0.9699, 0.0878,  ..., 0.1466, 0.9087, 0.4705],\n",
      "         [0.8170, 0.8187, 0.3619,  ..., 0.0927, 0.7144, 0.2004]]])\n"
     ]
    }
   ],
   "source": [
    "input_image = torch.rand(3,28,28)\n",
    "print(input_image.size())\n",
    "print(input_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nn.Flatten\n",
    "^^^^^^^^^^^^^^^^^^^^^^\n",
    "We initialize the `nn.Flatten  <https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html>`_ \n",
    "layer to convert each 2D 28x28 image into a contiguous array of 784 pixel values (\n",
    "the minibatch dimension (at dim=0) is maintained).\n",
    "^^^^^^^^^^^^^^^^^^^^^^\n",
    "nn.Flatten\n",
    "我们初始化nn.Flatten 层以将每个 2D 28x28 图像转换为 784 个像素值的连续数组（保持小批量维度（dim=0））。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 784])\n"
     ]
    }
   ],
   "source": [
    "flatten = nn.Flatten()\n",
    "flat_image = flatten(input_image)\n",
    "print(flat_image.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nn.Linear \n",
    "^^^^^^^^^^^^^^^^^^^^^^\n",
    "The `linear layer <https://pytorch.org/docs/stable/generated/torch.nn.Linear.html>`_\n",
    "is a module that applies a linear transformation on the input using its stored weights and biases.\n",
    "\n",
    "nn.Linear \n",
    "的线性层 是使用其存储的权重和偏置所述输入施加线性变换的模块。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 20])\n"
     ]
    }
   ],
   "source": [
    "layer1 = nn.Linear(in_features=28*28, out_features=20)\n",
    "hidden1 = layer1(flat_image)\n",
    "print(hidden1.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nn.ReLU\n",
    "^^^^^^^^^^^^^^^^^^^^^^\n",
    "Non-linear activations are what create the complex mappings between the model's inputs and outputs.\n",
    "They are applied after linear transformations to introduce *nonlinearity*, helping neural networks\n",
    "learn a wide variety of phenomena.\n",
    "\n",
    "In this model, we use `nn.ReLU <https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html>`_ between our\n",
    "linear layers, but there's other activations to introduce non-linearity in your model.\n",
    "神经网络ReLU\n",
    "非线性激活是在模型的输入和输出之间创建复杂映射的原因。它们在线性变换之后应用以引入非线性，帮助神经网络学习各种各样的现象。\n",
    "\n",
    "在这个模型中，我们在我们的线性层之间使用了nn.ReLU，但是还有其他的激活来在你的模型中引入非线性\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before ReLU: tensor([[-0.7128, -0.2602,  0.4458,  0.1306, -0.4329,  0.2058,  0.4162,  0.3180,\n",
      "         -0.1009,  0.0976, -0.1626,  0.4855, -0.0829,  0.1053, -0.1821,  0.5460,\n",
      "          0.2845, -0.4505,  0.1272,  0.1751],\n",
      "        [-0.4854,  0.1137,  0.7212, -0.0341, -0.4364,  0.4438, -0.0361,  0.2504,\n",
      "         -0.4727, -0.0468, -0.1793,  0.2106, -0.2118,  0.1037, -0.3460,  0.3326,\n",
      "          0.0329, -0.5909, -0.0587, -0.2260],\n",
      "        [-0.8123,  0.0973,  0.5791, -0.0712, -0.1970,  0.3900, -0.0823,  0.3412,\n",
      "          0.0440, -0.0662, -0.1100,  0.3236, -0.0063,  0.2001, -0.5269,  0.8877,\n",
      "          0.2186, -0.3165,  0.1869, -0.0445]], grad_fn=<AddmmBackward>)\n",
      "\n",
      "\n",
      "After ReLU: tensor([[0.0000, 0.0000, 0.4458, 0.1306, 0.0000, 0.2058, 0.4162, 0.3180, 0.0000,\n",
      "         0.0976, 0.0000, 0.4855, 0.0000, 0.1053, 0.0000, 0.5460, 0.2845, 0.0000,\n",
      "         0.1272, 0.1751],\n",
      "        [0.0000, 0.1137, 0.7212, 0.0000, 0.0000, 0.4438, 0.0000, 0.2504, 0.0000,\n",
      "         0.0000, 0.0000, 0.2106, 0.0000, 0.1037, 0.0000, 0.3326, 0.0329, 0.0000,\n",
      "         0.0000, 0.0000],\n",
      "        [0.0000, 0.0973, 0.5791, 0.0000, 0.0000, 0.3900, 0.0000, 0.3412, 0.0440,\n",
      "         0.0000, 0.0000, 0.3236, 0.0000, 0.2001, 0.0000, 0.8877, 0.2186, 0.0000,\n",
      "         0.1869, 0.0000]], grad_fn=<ReluBackward0>)\n"
     ]
    }
   ],
   "source": [
    "print(f\"Before ReLU: {hidden1}\\n\\n\")\n",
    "hidden1 = nn.ReLU()(hidden1)\n",
    "print(f\"After ReLU: {hidden1}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nn.顺序\n",
    "nn.Sequential是一个有序的模块容器。数据按照定义的相同顺序通过所有模块。您可以使用顺序容器来组合一个快速网络，例如seq_modules.\n",
    "\n",
    "\n",
    "nn.Sequential\n",
    "^^^^^^^^^^^^^^^^^^^^^^\n",
    "`nn.Sequential <https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html>`_ is an ordered \n",
    "container of modules. The data is passed through all the modules in the same order as defined. You can use\n",
    "sequential containers to put together a quick network like ``seq_modules``.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.2219, -0.0906, -0.1893,  0.3605,  0.4001,  0.0720,  0.1043,  0.0740,\n",
      "          0.0646,  0.1858],\n",
      "        [-0.2267, -0.0076, -0.1758,  0.2844,  0.5320,  0.0600,  0.0425,  0.0784,\n",
      "         -0.0612,  0.1324],\n",
      "        [-0.2325, -0.1708, -0.2231,  0.3062,  0.4053,  0.0145,  0.0477,  0.1550,\n",
      "         -0.0026,  0.1346]], grad_fn=<AddmmBackward>)\n"
     ]
    }
   ],
   "source": [
    "seq_modules = nn.Sequential(\n",
    "    flatten,\n",
    "    layer1,\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(20, 10)\n",
    ")\n",
    "input_image = torch.rand(3,28,28)\n",
    "logits = seq_modules(input_image)\n",
    "print(logits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nn.Softmax\n",
    "^^^^^^^^^^^^^^^^^^^^^^\n",
    "The last linear layer of the neural network returns `logits` - raw values in [-\\infty, \\infty] - which are passed to the\n",
    "`nn.Softmax <https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html>`_ module. The logits are scaled to values \n",
    "[0, 1] representing the model's predicted probabilities for each class. ``dim`` parameter indicates the dimension along \n",
    "which the values must sum to 1. \n",
    "\n",
    "\n",
    "nn.Softmax\n",
    "神经网络的最后一个线性层返回logits - [-infty, infty] 中的原始值 - 传递给 nn.Softmax模块。logits 被缩放到值 [0, 1]，代表模型对每个类的预测概率。dim参数指示值必须总和为 1 的维度。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.0728, 0.0830, 0.0752, 0.1304, 0.1356, 0.0977, 0.1009, 0.0979, 0.0970,\n",
      "         0.1095],\n",
      "        [0.0729, 0.0908, 0.0768, 0.1216, 0.1558, 0.0972, 0.0955, 0.0990, 0.0861,\n",
      "         0.1045],\n",
      "        [0.0743, 0.0790, 0.0750, 0.1273, 0.1406, 0.0951, 0.0983, 0.1095, 0.0935,\n",
      "         0.1073]], grad_fn=<SoftmaxBackward>)\n"
     ]
    }
   ],
   "source": [
    "softmax = nn.Softmax(dim=1)\n",
    "pred_probab = softmax(logits)\n",
    "print(pred_probab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model Parameters\n",
    "-------------------------\n",
    "Many layers inside a neural network are *parameterized*, i.e. have associated weights \n",
    "and biases that are optimized during training. Subclassing ``nn.Module`` automatically \n",
    "tracks all fields defined inside your model object, and makes all parameters \n",
    "accessible using your model's ``parameters()`` or ``named_parameters()`` methods. \n",
    "\n",
    "In this example, we iterate over each parameter, and print its size and a preview of its values. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Model structure: \", model, \"\\n\\n\")\n",
    "\n",
    "for name, param in model.named_parameters():\n",
    "    print(f\"Layer: {name} | Size: {param.size()} | Values : {param[:2]} \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Further Reading\n",
    "--------------\n",
    "- `torch.nn API <https://pytorch.org/docs/stable/nn.html>`_\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
